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(54) Image processing apparatus 

(57) This invention is an image processing appara- 
tus including a photographing condition estimation unit 
(13) for estimating a photographing condition of the in- 
put image on the basis of photometric information of a 
photometric evaluation unit (7) and focal information of 
a focal point detection unit (8); a Y/C separation unit (1 2) 
for separating the input image into a luminance signal 
and a color difference signal; a luminance correction unit 
(17) for extracting an edge from the luminance signal 
and correcting the luminance signal by a gradation con- 
version curve obtained by weighting depending on the 
photographing condition; a color difference correction 



unit (1 8) for performing correction on the color difference 
signal on the basis of luminance signals obtained before 
and after the gradation correction and a theoretical limit 
characteristic of color reproduction; a skin color correc- 
tion unit (41) for performing skin color correction or the 
like depending on the photographing condition; a Y/C 
synthesis unit (1 9) for synthesizing the luminance signal 
and the color difference signal obtained afterthe correc- 
tion. The image processing apparatus performs grada- 
tion correction on the input image such that appropriate 
hue and saturation are achieved while emphasizing a 
main object. 
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Description 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0001] The present invention relates to an image 
processing apparatus and, more specifically, to an im- 
age processing apparatus for adjusting a gradation 
range of an input image. 

2. Related Art Statement 

[0002] In an image processing apparatus for synthe- 
sizing a plurality of images picked up under different ex- 
posure conditions to generate one wide dynamic range 
image, a technique for adjusting a gradation range is 
used. For example, in Japanese Patent Application No. 
11-338551 which has not been published yet, an image 
process device for dividing each of images into an ap- 
propriate exposure region and an inappropriate expo- 
sure region, performing gradation correction for each 
appropriate exposure region, and synthesizing the ap- 
propriate exposure regions of the images subjected to 
the gradation correction with each other to generate one 
wide dynamic range image is disclosed. In addition, as 
an example of an apparatus to which the image process- 
ing apparatus is applied, a super latitude digital camera 
being capable of picking up an image of an object in a 
wider dynamic range is described. 
[0003] A gradation converting process in the image 
processing apparatus described above is performed on 
the basis of histogram flattening of an edge portion. This 
histogram flattening is a technique which is on the as- 
sumption that a main object as a large number of edges, 
the other portions such as a background has a small 
number of edges. 

[0004] On the other hand, in a conventional digital 
camera, a color difference signal is also converted on 
the basis of a coeff icient obtained when a luminance sig- 
nal is subjected to gradation conversion. More specifi- 
cally, it is assumed that a luminance signal Y org is con- 
verted by a gradation conversion characteristic F such 
that Y tra = F (Y org ) is satisfied. In this case, convention- 
ally, a conversion coefficient gain of a luminance signal 
is calculated by: 

ga'»n = Y tra /Y org , 



and 

the conversion coefficient is directly used to convert 
color difference signals as follows: 



Cb^ = gain • Cb org 



Cr tra = gain • Cr org . 

[0005] In a digital camera, a bit count obtained when 
5 an image signal output as an analog signal from a CCD 
into a digital signal is often a bit count obtained when 
the image signal is processed and then recorded on, e. 
g., a recording medium. 

[0006] In this case, even though a single input image 
10 is handled to perform conversion of an amount of infor- 
mation, especially, a reduction of an amount of informa- 
tion, the gradation range of the image may have to be 
adjusted. 

[0007] The technique of gradation correction is popu- 

15 larly handled in various devices such as a printer device 
or a monitor device which is not limited to the digital cam- 
era and processes an image. The above technique can 
be applied to not only a case in which a plurality of im- 
ages are handled but also a case in which a single input 

20 image is handled. 

[0008] The technique of histogram flattening on the 
assumption that the main object as described above has 
a large number of edges can correspond to an object in 
a relatively wide range. However, the technique cannot 

25 completely correspond to the object. As an example 
which may correspond to the exception, a case in which 
a person is photographed in a relatively small size on a 
background having a plurality of shapes or contours is 
used. At this time, the background is decided as a main 

30 object to detect a large number of edges from the back- 
ground portion, and a gradation range which is assigned 
to a person is reduced. 

[0009] In a technique which performs gradation con- 
version to the color difference signal as described above 
35 by using a conversion coefficient equal to the conver- 
sion coefficient of a luminance signal, unnatural colors 
may be generated by a high-luminance portion. More 
specifically, color reproduction in a color space (e.g., Y, 
Cb, and Cr space) has a theoretical limit characteristic 
40 (see FIG. 11 showing an embodiment of the present in- 
vention). The theoretical limit characteristic of the color 
reproduction is a characteristic in which a color differ- 
ence range in which colors can be reproduced increases 
as a luminance Y is increased, and a color difference 
45 range in which colors can be reproduced when the lu- 
minance exceeds the luminance Y. More specifically, a 
color reproduction range is narrow because the colors 
become blackish as a whole at a low luminance, colors 
in a wide range can be reproduced at an appropriate 
so luminance, and a color reproduction range becomes 
narrow again because the colors becomes whitish as a 
whole at a high luminance. 

[0010] When gradation conversion is performed like 
the case of luminance without consideration of the color 
55 reproduction range, the color reproduction range is 
close to or exceeds the limit of the color reproduction 
range, and colors after the gradation conversion may be 
whitish. In order to cope with the problem, a process of 
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suppressing a saturation of a high-luminance portion is 
added in a conventional technique. However, it is not 
sufficient, and the realization of a technique of more im- 
proving color reproduction is desired. 

OBJECTS AND SUMMARY OF THE INVENTION 

[0011] It is an object of the present invention to pro- 
vide an image processing apparatus being capable of 
adaptively adjusting a gradation range of a main object 
in accordance with a photographic scene. 
[0012] It is another object of the present invention to 
provide an image processing apparatus being capable 
of adjusting a more optimum saturation in consideration 
of the theoretical limit characteristic of color reproduc- 
tion. 

[0013] In short, the present Invention is an Image 
processing apparatus, for adjusting a gradation range 
of an input image, including: photographing condition 
estimation means for estimating a photographing con- 
dition of the input image; and gradation correction 
means for performing gradation correction on the input 
image on the basis of the photographing condition to ad- 
just the gradation range to a predetermined gradation 
range. 

[0014] The present invention is an image processing 
apparatus, for adjusting a gradation range of an input 
image, including: luminance-color difference separation 
means for separating the input image into a luminance 
signal and a color difference signal; gradation correction 
means for performing gradation correction on the lumi- 
nance signal to adjust the gradation range to a prede- 
termined gradation range; color difference correction 
means for correcting the color difference signal on the 
basis of a luminance signal obtained before the grada- 
tion correction and output from the luminance-color dif- 
ference separation means, a luminance signal obtained 
after the gradation correction, and a theoretical limit 
characteristic of color reproduction; and luminance- 
color difference synthesis means for synthesizing the lu- 
minance signal obtained after the gradation correction 
and a color difference signal obtained after the correc- 
tion into an image signal. 

[0015] Furthermore, the present invention is an image 
processing apparatus for adjusting a gradation range of 
an input image, including: photographing condition es- 
timation means for estimating a photographing condi- 
tion of the input image; luminance-coior difference sep- 
aration means for separating the input image into a lu- 
minance signal and a color difference signal; gradation 
correction means for performing gradation correction on 
the luminance signal on the basis of the photographing 
condition to adjust the gradation range to a predeter- 
mined gradation range; color difference correction 
means for performing correction on the color difference 
signal on the basis of a luminance signal obtained be- 
fore the gradation correction and output from the lumi- 
nance-color difference separation means, a luminance 



signal obtained after gradation correction and output 
from the gradation correction means, and a theoretical 
limit characteristic of color reproduction; and luminance- 
color difference synthesis means for synthesizing an im- 
5 age signal on the basis of the luminance signal obtained 
after the gradation correction and a color difference sig- 
nal obtained after the correction. 
[0016] The present invention is an image processing 
apparatus for processing an image group constituted by 
10 a plurality of images obtained by image pickup to the 
same object under different exposure conditions to gen- 
erate one wide dynamic range image, including: photo- 
graphing condition estimation means for estimating a 
photographing condition; extraction means for extract- 
15 jng an appropriate exposure region on the basis of an 
image signal level of each image in the image group; 
gradation correction means for performing gradation 
correction on the appropriate exposure region on the ba- 
sis of the photographing condition; and synthesis means 
20 for synthesizing appropriate exposure regions subject- 
ed to gradation correction by the gradation correction 
means to generate one wide dynamic range image. 
[0017] In addition, the present invention is an image 
processing apparatus for processing an image group 
25 constituted by a plurality of images obtained by image 
pickup to the same object under different exposure con- 
ditions to generate one wide dynamic range image, in- 
cluding: luminance-color difference separation means 
for separating an image signal of each image in the im- 
30 age group into a luminance signal and a color difference 
signal; extraction means for extracting an appropriate 
exposure region on the basis of a signal level of the lu- 
minance signal; gradation correction means for per- 
forming gradation correction on a luminance signal of 
35 the appropriate exposure region; color difference cor- 
rection means for performing correction on a color dif- 
ference signal of the appropriate exposure region on the 
basis of a luminance signal obtained before the grada- 
tion correction and output from the luminance-color dif- 
40 ference separation means, a luminance signal obtained 
after gradation correction and output from the gradation 
correction means, and a theoretical limit characteristic 
of color reproduction; luminance-color difference syn- 
thesis means for synthesizing an image signal on the 
45 basis of the luminance signal obtained after the grada- 
tion correction and a color difference signal obtained af- 
ter the correction; and synthesis means for synthesizing 
the synthesized image signal of the appropriate expo- 
sure regions to generate one wide dynamic range im- 
so age. 

[0018] The present invention is an image processing 
apparatus for processing an image group constituted by 
a plurality of images obtained by image pickup to the 
same object under different exposure conditions to gen- 
55 erate one wide dynamic range image, including: photo- 
graphing condition estimation means for estimating a 
photographing condition; luminance-color difference 
separation means for separating an image signal of 



3 



EP 1 128 660 A2 



6 



each image in the image group into a luminance signal 
and a color difference signal; extraction means for ex- 
tracting an appropriate exposure region on the basis of 
a signal level of the luminance signal; gradation correc- 
tion means for performing gradation correction on the 
luminance signal of the appropriate exposure region on 
the basis of the photographing condition; color differ- 
ence correction means for performing correction on a 
color difference signal of the appropriate exposure re- 
gion on the basis of a luminance signal obtained before 
the gradation correction and output from the luminance- 
color difference separation means, a luminance signal 
obtained after gradation correction and output from the 
gradation correction means, and a theoretical limit char- 
acteristic of color reproduction; luminance-color differ- 
ence synthesis means for synthesizing an image signal 
on the basis of the luminance signal obtained after the 
gradation correction and a color difference signal ob- 
tained afterthe correction; and synthesis means for syn- 
thesizing the synthesized image signal of the appropri- 
ate exposure regions to generate one wide dynamic 
range image. 

[0019] In addition, the present invention is a recording 
medium on which a process program executed by a 
computer is recorded, wherein the process program ad- 
justs a gradation range of an input image and includes 
the step of separating the input image into a luminance 
signal and a color difference signal; the step of perform- 
ing gradation correction on the luminance signal to ad- 
just the gradation range to a predetermined gradation 
range; the step of performing correction on the color dif- 
ference signal on the basis of a luminance signal ob- 
tained before the gradation correction, a luminance sig- 
nal obtained after the gradation correction, and a theo- 
retical luminance characteristic of color reproduction; 
and the step of synthesizing an image signal on the ba- 
sis of the luminance signal obtained after gradation cor- 
rection and a color difference signal obtained after the 
correction. 

[0020] The present invention is a recording medium 
on which a process program executed by a computer is 
recorded, wherein the process program processes an 
image group constituted by a plurality of images ob- 
tained by image pickup performed to the same object 
under different exposure conditions to generate one 
wide dynamic range image and includes: the step of 
separating an image signal of each image in the image 
group into a luminance signal and a color difference sig- 
nal; the step of extracting an appropriate exposure re- 
gion on the basis of a signal level of the luminance sig- 
nal; the step of performing gradation correction on a lu- 
minance signal of the appropriate exposure region: the 
step of performing correction on a color difference signal 
of the appropriate exposure region on the basis of a lu- 
minance signal obtained before the gradation correc- 
tion, a luminance signal obtained after gradation correc- 
tion, and a theoretical limit characteristic of color repro- 
duction; the step of synthesizing an image signal on the 



basis of the luminance signal obtained after the grada- 
tion correction and a color difference signal obtained af- 
ter the conection; and the step of synthesizing the syn- 
thesized image signal of the appropriate exposure re- 
5 gions to generate one wide dynamic range image. 
[0021] These objects and advantages of the present 
invention will become further apparent from the follow- 
ing detailed explanation. 



10 BRIEF DESCRIPTION OF THE DRAWINGS 

[0022] FIG. 1 is a block diagram showing a basic con- 
figuration of an electronic camera according to the first 
embodiment of the present invention. 
is [0023] FIG. 2 is a block diagram showing a detailed 
configuration of a photographing condition estimation 
unit according to the first embodiment. 
[0024] FIG. 3 is a block diagram showing a detailed 
configuration of a gradation correction unit according to 
20 the first embodiment. 

[0025] FIG. 4 is a diagram showing an example of a 
divisional pattern for an evaluation photometric opera- 
tion in the first embodiment. 

[0026] FIG. 5 is a table showing a classification pat- 
25 tern of scenes from AF information and AE information 
in the first embodiment. 

[0027] FIGS. 6A to 6F are diagrams showing weight 
coefficients obtained in edge histogram calculation on 
the basis of the classification pattern shown in FIG. 5 in 
30 the first embodiment. 

[0028] FIG. 7 is a flow chart showing an image con- 
version process in the first embodiment. 
[0029] FIG. 8 is a block diagram showing a basic con- 
figuration of an electronic camera according to the sec- 
35 ond embodiment of the present invention. 

[0030] FIG. 9 is a block diagram showing a detailed 
configuration of a luminance correction unit according 
to the second embodiment. 

[0031] FIG. 1 0 is a block diagram showing a detailed 
40 configuration of a color difference correction unit ac- 
cording to the second embodiment. 
[0032] FIG. 11 is a graph showing a manner for per- 
forming color difference correction in consideration of a 
theoretical limit characteristic of color reproduction in 
45 the second embodiment. 

[0033] FIG. 1 2 is a flow chart showing an image con- 
version process in the second embodiment. 
[0034] FIG. 13 is a block diagram showing a basic 
configuration of an electronic camera according to the 
so third embodiment of the present invention. 

[0035] FIG. 14 is a flow chart showing an image con- 
version process in the third embodiment. 
[0036] FIG. 15 is a block diagram showing a basic 
configuration of an electronic camera according to the 
55 fourth embodiment of the present invention. 

[0037] FIG. 1 6 is a block diagram showing a detailed 
configuration of a photographing condition estimation 
unit according to the fourth embodiment. 
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[0038] FIG. 17 is a block diagram showing a detailed 
configuration of a conversion characteristic calculation 
unit according to the fourth embodiment. 
[0039] FIG. 1 8 is a flow chart showing an image con- 
version process according to the fourth embodiment. 
[0040] FIG. 19 is a block diagram showing a basic 
configuration of an electronic camera according to the 
fifth embodiment of the present invention. 
[0041] FIG. 20 is a block diagram showing a detailed 
configuration of a luminance correction unit according 
to the fifth embodiment. 

[0042] F1G. 21 is a block diagram showing a detailed 
configuration of a color difference correction unit ac- 
cording to the fifth embodiment. 

[0043] FIG. 22 is a block diagram showing a basic 
configuration of an electronic camera according to the 
sixth embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0044] Embodiments of the present invention will be 
described below with reference to the accompanying 
drawings. 

[0045] FIGS. 1 to 7 show the first embodiment of the 
present invention, wherein FIG. 1 is a block diagram 
showing a basic configuration of an electronic camera. 
[0046] This embodiment is obtained by applying an 
image processing apparatus for adjusting a gradation 
range according to the present invention to an electronic 
camera. 

[0047] This electronic camera comprises: a CCD 4 
constituted by a color CCD or the like of a single CCD 
type having an electronic shutter function or the like and 
for photoelectrically converting an object image as an 
image signal; a lens system 1 for focusing the object 
image on the CCD 4; a diaphragm 2 for controlling a 
passing range of a flux of light passing through the lens 
system 1 ; a low-pass filter 3 constituted by an optical 
filter for removing an unnecessary high-frequency com- 
ponent from the flux of light passing through the dia- 
phragm 2; an A/D converter 5 for converting an analog 
image signal which is output from the CCD 4, from which 
a noise component is removed by a correlative duplex 
sampling circuit or the like (not shown), and which is 
subjected to amplification into a digital signal; an image 
buffer 6 for storing image data of one screen digitized 
by the AID converter 5; a photometric evaluation unit 7 
for reading image data from the image buffer 6 to calcu- 
late a luminance distribution and for controlling the ap- 
erture diameter of the diaphragm 2 and the electronic 
shutter of the CCD 4 such that an appropriate exposure 
is obtained in a photographing state; a focal point de- 
tection unit 8 for reading image data from the image buff- 
er 6 to detect a focal position and for controlling an AF 
motor 9 (to be described later) on the basis of the de- 
tection result; an AF motor 9 controlled by the focal point 
detection unit 8 to drive an AF lens of the lens system 



1 and for focusing an object image on the CCD 4; an 
interpolation u nit 1 0 for interpolating image data of a sin- 
gle CCD read from the image buffer 6 and converting 
tree-CCD image data; an operation buffer 11 for storing 
5 the image data after interpolation; a Y/C separation unit 
12 serving as a luminance-color difference separation 
means for separating three-CCD image data read from 
the operation buffer 11 into a luminance signal Y and 
color difference signals Cb and Cr; a photographing 
10 condition estimation unit 13 serving as a photographing 
condition estimation means for estimating a photo- 
graphing condition (as will be described below) on the 
basis of photometric information output from the photo- 
metric evaluation unit 7 and focal information output 
is from the focal point detection unit 8; a gradation correc- 
tion unit 14 serving as a gradation correction means for 
reading a luminance signal Y from the Y/C separation 
unit 12 to extract an edge component, for performing 
weighting used when the histogramof an edge which is 
20 an amount of information is calculated with reference to 
the estimation result obtained by the photographing 
condition estimation unit 13 to calculate a conversion 
characteristic, and for performing gradation conversion 
of the luminance signal Y and the color difference sig- 
25 nals Cb and Cr output from the Y/C separation unit 1 2; 
an output unit 1 5 for converting an image the gradation 
range of which is adjusted by the gradation correction 
unit 14 into the original RGB signal or the like and then 
outputting the RGB signal to, e.g., a recording medium 
30 or a display device; and a control unit 16 for receiving 
detection results of the photometric evaluation unit 7 
and the focal point detection unit 8 and controlling the 
entire electronic camera including the interpolation unit 
10, the photographing condition estimation unit 1 3. and 
35 the gradation correction unit 1 4. 

[0048] FIG. 2 is a block diagram showing the detailed 
configuration of the photographing condition estimation 
unit 13. 

[0049] The focal (AF) information output from the fo- 
40 cal point detection unit 8 and input through the control 
unit 1 6 is input to a focal position estimation unit 20 serv- 
ing as a focal position estimation means and classified 
into one of three types (see FIG. 5), e.g., a scenic pho- 
tographing operation (5 m to <»), a portraiture photo- 
45 graphing operation (1 m to 5 m), and a close-up photo- 
graphing operation (1 m or less) according to an object 
distance. 

[0050] The photometric (AE) information output from 
the photometric evaluation unit 7 and input through the 

so control unit 1 6 is input to an object distribution estimation 
unit 21 serving as an object distribution estimation 
means, and the luminance distributions of the photomet- 
ric information are classified into several types. 
[0051] More specifically, the photometric evaluation 

55 unit 7 classifies regions on the CCD 4 into 1 3 regions 
as shown in, e.g., FIG. 4, to perform a divisional photo- 
metric operation. FIG. 4 is a diagram showing a division- 
al pattern for an evaluation photometric operation. 
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[0052J That is, the central region, the left adjacent re- 
gion, and the right adjacent region of the centermost 
portion are represented by a 1f a 2 , and a 3 , respectively. 
[0053] In addition, the upper and lower regions of the 
region a n in an inner peripheral portion surrounding the 
centermost portion are represented by a 4 and a 5 , the 
left and right regions of the region a 4 are represented 
by a 6 and a 7 , and the left and right regions of the region 
a 5 are represented by a 8 and a 9 . 
[0054] The upper left region, the upper right region, 
and the lower left region, and the lower right region in 
an outer peripheral portion surrounding the innerperiph- 
eral portion are represented by a 10 , a l1t a 12 , and a 13 . 
[0055] In a divisional photometric operation using 
these regions, the object distribution estimation unit 21 
calculates the following evaluation parameters: 

[Equation 1] 
S n = |a 2 -a 3 | 



[Equation 2] 
S 2 = max(|a 4 -a 6 |, |a 4 -a 7 |) 



[Equation 3] 
S 3 = max(a 10 , a 1T ) - Av 
Av= (Xai)/13 

[0056] More specifically, the evaluation parameter S., 
indicates the difference between the luminances of the 
left and right regions of the centermost portion, the eval- 
uation parameter S 2 indicates a large one of the differ- 
ences between the luminances of the upper central re- 
gion and the upper left and upper right regions of the 
inner peripheral portion, and the evaluation parameter 
S 3 indicates the difference between a large one of the 
luminances of the upper left and upper right regions of 
the outer peripheral portion and an average luminance 
of an entire screen. 

[0057] These evaluation parameters are obtained 
from the object distribution estimation unit 21 , and clas- 
sification of the focal positions is obtained by the focal 
position estimation unit 20, so that integral classification 
as shown in FIG. 5 is performed in the integration unit 
22 serving as an integration means. FIG. 5 is a table 
showing a classification pattern of scenes on the basis 
of AF information and AE information. 
[0058] As shown in FIG. 5, when the AF information 
indicates 5mto«,a scenic photographing operation is 
defined, and the evaluation parameter S 3 is compared 
with a predetermined value Th-,. At this time, when the 
evaluation parameter S 3 is larger than the predeter- 



mined value Th<, -, at least one of the regions a 10 and a t1 
has a luminance which is equal to or larger than the av- 
erage luminance of the entire screen to some extent. 
For this reason, the photographing operation is deter- 
5 mined as a photographing operation for a landscape 
containing sky on the upper side (Type 1 ). On the other 
hand, when the evaluation parameter S 3 is smaller than 
the predetermined value Th n , the photographing oper- 
ation is determined as a photographing operation for a 
io : landscape containing no sky on the upper side or a land- 
scape containing slight sky on the upper side (Type 2). 
[0059] When the AF information indicates 1 m to 5 m, 
a portraiture photographing operation is determined, 
and the evaluation parameter S 2 is compared with a pre- 
15 determined value Th 2 . At this time, when the evaluation 
parameter S 2 is larger than the predetermined value 
Th 2> the photographing operation is determined as a 
photographing operation for a portrait of a single person 
(Type 3). When the evaluation value S 2 is smaller than 
20 the predetermined value Th 2 , the photographing oper- 
ation is determined as a photographing operation for a 
portrait of plural persons (Type 4). 
[0060] In addition, when the AF information indicates 
1 m or less, the photographing operation is determined 
25 as a close-up photographing operation, and the evalu- 
ation parameter is compared with a predetermined 
value Th 3 . At this time, when the evaluation parameter 
S n is larger than the predetermined value Th 3 , the pho- 
tographing operation is determined as a close-up pho- 
30 tographing operation for a single subject (Type 5). On 
the other hand, when the evaluation parameter is 
smaller than the predetermined value Th 3 , the photo- 
graphing operation is a close-up photographing opera- 
tion for plural subjects (Type 6). 
35 [0061 ] Results obtained by classifying the scenes into 
these types are output from the integration unit 22 to the 
gradation correction unit 14. 

[0062] Subsequently, FIG. 3 is a block diagram show- 
ing a detailed configuration of the gradation correction 
40 unit 14. 

[0063] When a luminance signal Y from the Y/C sep- 
aration unit 12 is input to the edge detection unit 26 serv- 
ing as a characteristic amount calculation means, an 
edge detection unit 26 performs edge detection on the 
45 basis of the control of the control unit 16. More specifi- 
cally, the edge detection unit 26 is constituted by a gen- 
eral edge detection operator such as a Laplacian or a 
Sobel. When the strength obtained by the edge detec- 
tion operator is equal to or larger than a predetermined 
so threshold value, binary information representing that an 
edge exists at a reference position is output. Otherwise, 
binary information representing that no edge exists at 
the reference position. 

[0064] On the other hand, when the results classified 
55 into types by the photographing condition estimation 
unit 13 are input to a pattern selection unit 24 serving 
as a selection means, the pattern selection unit 24 
based on the control unit 1 6 selects a weight pattern de- 
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pending on a type Irom a weight pattern ROM 25 in 
which a plurality of weight patterns as shown in FIG. 6A 
to 6F are stored in advance. FIGS. 6A to 6F are dia- 
grams showing weight coefficients obtained in histo- 
gram calculation based on the classification pattern 
shown in FIG. 5. FIG. 6A shows a weight pattern corre- 
sponding to the Type 1 , FIG. 6B shows a weight pattern 
corresponding to the Type 2, FIG. 6C shows a weight 
pattern corresponding to the Type 3, FIG. 6D shows a 
weight pattern corresponding to the Type 4, FIG. 6E 
shows a weight pattern corresponding to the Type 5, and 
FIG. 6F shows a weight pattern corresponding to the 
Type 6. 

[0065] In this manner, a histogram generation unit 27 
serving as a histogram generation means calculates an 
edge histogram representing an appearance frequency 
for luminance levels with respect to pixels constituting 
an edge and pixels near these pixels on the basis of the 
results output from the edge detection unit 26. When the 
histogram is generated, the histogram is calculated 
such that weight corresponding to pixel positions in an 
image as shown in FIGS. 6A to 6F. In addition, in the 
histogram generation unit 27, the calculated edge his- 
togram is integrated to be converted into an accumulat- 
ed edge histogram. 

[0066] In a conversion curve calculation unit 28 serv- 
ing as a gradation conversion curve calculation means, 
an edge histogram is convoluted by using a gaussian 
kernel or the like to generate a target histogram. By us- 
ing the target histogram and the accumulated edge his- 
togram output from the histogram generation unit 27, a 
tone curve serving as a gradation correction character- 
istic is calculated. 

[0067] The conversion unit 29 serving as a conversion 
means performs gradation correction based on a tone 
curve obtained by the conversion curve calculation unit 
28 to image data input from the Y/C separation unit 12, 
and image data after the gradation correction is output 
to an output unit 15. In this conversion unit 29, gradation 
correction of the luminance signal Y is performed, and 
gradation corrections of the color difference signals Cb 
and Cr are sequentially performed. The obtained signals 
are output to the output unit 15. 

[0068] The output unit 1 5 receives the luminance sig- 
nal Y and the color difference signals Cb and Cr sub- 
jected to gradation correction on generate, e.g., an orig- 
inal RGB signal and to output the original RGB signal. 
[0069] FIG. 7 is a flowchart showing an image con- 
version process. 

[0070] An object image focused on the CCD 4 consti- 
tuted by a single CCD is converted into an image signal 
by a photographing operation to output the image signal, 
[0071] This image signal is converted into a digital sig- 
nal by the A/D converter 5 and then temporarily stored 
in the image buffer 6. 

[0072] The photometric evaluation unit 7 and the focal 
point detection unit 8 output AE information and AF in- 
formation to the control unit 1 6, as described above, on 



the basis of the image data accumulated in the image 
buffer 6 (step S1). 

[0073] On the other hand, the image data stored in 
the image buffer 6 is transmitted to the interpolation unit 

5 10 and interpolated with respect to R image data, G im- 
age data, and B image data. The image data is convert- 
ed into three-CCD image data (step S2), and the three- 
CCD image data is stored in the operation buffer 11 . 
[0074] In the Y/C separation unit 1 2, RGB image data 

10 is read from the operation buffer 11 to calculate the lu- 
minance signal Y and the color difference signals Cb 
and Cr as expressed by the following Equation 4 (step 
S3): 



15 



20 



[Equation 4] 
Y = 0.29900R + 0.58700G + 0.14400B 
Cb = -0.16874R - 0.331 26G + 0.50000B 
Cr = 0.50000R - 0.41 869G - 0.081 31 B 



[0075] The gradation correction unit 14, in the edge 
25 detection unit 26, operates a known quadratic differen- 
tial filter such as a Laplacian to the luminance signal Y 
to extract an edge component (step S4) . A threshold val- 
ue which is about twice a standard deviation is set for 
the extracted edge component to perform a binarizing 
30 process (step S5). 

[0076] On the other hand, in the photographing con- 
dition estimation unit 13, the above photographing con- 
dition is estimated on the basis of the AF information 
and the AE information (step S6), and one of Type 1 to 
35 Type 6 corresponding to a weight pattern is selected 
(step S7). A weight coefficient corresponding to the se- 
lected weight pattern as shown in FIGS. 6A to 6F is read 
from the weight pattern ROM 25 (step S8). 
[0077] An edge histogram is formed by the histogram 
40 generation unit 27 on the basis of the edge component 
binarized in step S5 and the weight pattern read in step 
S8 (step 59). From the edge histogram, an accumulated 
edge histogram is generated (step S10). 
[0078] On the basis of the edge histogram obtained 
45 as described above, the conversion curve calculation 
unit 28 calculates a gradation conversion curve (step 
S11). 

[0079] In the subsequent conversion unit 29, the lu- 
minance signal Y and the color difference signals Cb 

so and Cr output from the Y/C separation unit 12 are sub- 
jected to a conversion process by the gradation conver- 
sion curve obtained by the conversion curve calculation 
unit 28 (step S12), so that the converted image data is 
output (step S13). 

55 [0080] Although both the photometric information and 
the focal information are used to estimate the photo- 
graphing conditions in the above description, the pho- 
tographing condition may be estimated by using only 
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one of the photometric information and the focal infor- 
mation to change weighting. If not only the photometric 
information and the focal information, but also one or 
more of zoom position information, multi-spot photomet- 
ric information, line-of-sight input information, emission 
information of an strobe flash, information of a detection 
sensorfor detecting the vertical and horizontal positions 
of an electronic camera, and white balance information 
are referred to, a photographing condition can be more 
exactly estimated. 

[0081] In addition, the above described technique of 
gradation correction depending on the photographing 
condition is applied to not only a color image but also a 
black-and-white image. 

[0082] In this first embodiment, gradation correction 
depending on a photographing condition is performed 
by the image processing apparatus constituted as a cir- 
cuit in an electronic camera. However, these processes 
can also be performed by a process program of a com- 
puter. In this case, photographing information such as 
photometric information and focal information is record- 
ed on, e.g., a header portion of an image file in a com- 
puter.a photographing condition is estimated on these 
pieces of photographing information, so that gradation 
correction appropriate to the photographing condition 
may be performed. 

[0083] The image processing apparatus is not only 
applied to an electronic camera, but also popularly ap- 
plied to devices such as printer devices which handle 
images. 

[0084] According to the first embodiment, a photo- 
graphing condition is decided on the basis of photo- 
- graphing information such as focal information and pho- 
tometric information, and weighting depending on the 
photographing condition is performed when an edge 
histogram is formed. For this reason, gradation correc- 
tion which is maximally appropriate to a photographing 
scene can be performed in consideration of a main ob- 
ject. 

[0085] FIGS. 8 to 12 show the second embodiment of 
the present invention, wherein FIG. 8 is a block diagram 
showing a basic configuration of an electronic camera, 
FIG. 9 is a block diagram showing a detailed configura- 
tion of a luminance correction unit, FIG. 10 is a block 
diagram showing a detailed configuration of a color dif- 
ference correction unit, FIG. 11 is a graph showing a 
manner for performing color difference correction in 
consideration of a theoretical limit characteristic of color 
reproduction, and FIG. 12 is a flow chart showing an im- 
age conversion process. 

[0086] The same reference numerals as in the first 
embodiment described above denote the same parts in 
the second embodiment, and a description thereof will 
be omitted. Only different points will be mainly described 
below. 

[0087] In the second embodiment, the luminance sig- 
nal Y and the color difference signals Cb and Cr sepa- 
rated by the Y/C separation unit 12 are input to a lumi- 



nance correction unit 17 serving as a gradation correc- 
tion means and a color difference correction unit 18 
serving as a color difference correction means, respec- 
tively. 

5 [0088] The luminance correction unit 17, as shown in 
FIG. 9, receives the luminance signal Y output from the 
Y/C separation unit 12, and performs gradation correc- 
tion for luminance. The luminance correction unit 17 is 
constituted by an edge extraction unit 30 serving as 
10 characteristic amount calculation means, a histogram 
generation unit 31 serving as a histogram generation 
means, a conversion curve calculation unit 32 serving 
as a gradation conversion curve calculation means, and 
a luminance conversion unit 33 serving as a luminance 
is conversion means. 

[0089] The processes in the luminance correction unit 
17 will be described with reference to FIG. 12. 
[0090] The luminance correction unit 1 7 reads the lu- 
minance signal Y output from the Y/C separation unit 12 
20 (step S21 ), the edge extraction unit 30 operates a filter 
such as a Laplacianto extract an edge component (step 
S22). The edge component is compared with a prede- 
termined threshold value with respect to pixels, and a 
binarizing process for determining whether a pixel is an 
25 edge or not (step S23). 

[0091] On the basis of information output from the 
edge extraction unit 30, the histogram generation unit 
31 generates an edge histogram representing an ap- 
pearance frequency of an edge to a luminance (step 
30 S24), and the edge histogram is integrated to generate 
an accumulated edge histogram (step S25). 
[0092] The conversion curve calculation unit 32 cal- 
culate a tone curve serving as a gradation correction 
characteristic as described above by using the accumu- 
35 lated edge histogram output from the histogram gener- 
ation unit 31 (step S26). 

[0093] The luminance conversion unit 33 performs 
gradation conversion to the luminance signal Y on the 
bas is of the conversion curve under the control of the 
40 control unit 1 6 (step S27), outputs the converted lumi- 
nance signal Y to the color difference correction unit 1 8, 
and outputs the converted luminance signal Y to a Y/C 
synthesis unit 19 serving as a luminance-color differ- 
ence synthesis means (step S28). 
45 [0094] In this manner, it is assumed that a luminance 
signal obtained before the gradation correction and out- 
put from the Y/C separation unit 12 is represented by 



Y and that a luminance signal subjected to after the 
gradation correction by the luminance conversion unit 
so 33 is represented by Y^. In this case, the luminance 
signals Y org and Y^, as described below, are used when 
gradation of a color difference is corrected in the color 
difference correction unit 1 8. 

[0095] The color difference correction unit 18, as 
55 showing in FIG. 1 0, the color difference signals Cb and 
Cr output from the Y/C separation unit 1 2 to perform gra- 
dation correction on a color difference. The color differ- 
ence correction unit 1 8 is constituted by a first correction 
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coefficient calculation unit 37 serving as a first calcula- 
tion means, a second correction coefficient calculation 
unit 35 serving as a second calculation means, a color 
reproduction limit characteristic ROM 36, and a color dif- 
ference conversion unit 38 serving as a color difference 
conversion means. 

[0096] In the color difference correction unit 18, the 
first correction coefficient calculation unit 37 receives a 
luminance signal Y org obtained before gradation correc- 
tion from the Y/C separation unit 12, and calculates a 
color reproduction range b org corresponding to the lumi- 
nance signal Y org as expressed in thefollowing Equation 
5 (step S31): 

[Equation 5] 

bor 9 = B(Y OIg ) 

[0097] In this equation, a function B(Y) is a function 
representing a theoretical limit characteristic of color re- 
production in a color space (Y, Cb, and Cr space), and 
has the following characteristic. That is s for example, as 
typically shown in FIG. 11, a color difference range in 
which color reproduction can be performed is widened 
as the luminance Y is increased, and the color difference 
range in which color reproduction can be performed is 
narrowed when the luminance Y exceeds a certain lu- 
minance. 

[0098] The calculation as described in Equation 5 is 
performed such that a color reproduction range b org cor- 
responding to the luminance signal Y org is calculated 
with reference to table data or the like stored in the color 
reproduction limit characteristic ROM 36 (step S30). 
[0099] The color reproduction limit characteristic 35 
ROM 36 stores the function B(Y) representing the the- 
oretical limit characteristic of color reproduction as table 
data in advance. In this case, the table data which is 
changed into a ROM in consideration of a load, a 
processing rate, and the like obtained by calculation is 
used. However, the table data may be obtained by ac- 
tual calculation. 

[0100] The second correction coefficient calculation 
unit 35 receives the luminance signal Y tra obtained after 
gradation correction and output from the luminance cor- 
rection unit 17 to calculate a color reproduction range 
b tra corresponding to the luminance signal Y^ as de- 
scribed in thefollowing Equation 6 which is similarto the 
Equation 5 described above (step S32): 

[Equation 6] 
b tra = B(Y tra ) 

[0101] The calculation described in Equation 6, simi- 
larly, is performed such that the color reproduction range 
b tra corresponding to the luminance Y^ is calculated 
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with reference to table data or the like stored in the color 
reproduction limit characteristic ROM 36 (step S30). 
[01 02] The color difference conversion unit 38 calcu- 
lates a conversion coefficient gain c corresponding to a 
color difference signal on the basis of the b org serving 
as a first correction coefficient and the b^ serving as a 
second correction coefficient as described in the follow- 
ing Equation 7: 

[Equation 7] 
gain c = b^/b^ 

[0103] In this manner, the conversion coefficient gain c 
corresponding to the color difference signal is defined 
as a ratio of the theoretical limit characteristic b org of 
color reproduction in the luminance signal Y org obtained 
before gradation correction on the theoretical limit char- 
acteristic b tra of color reproduction in the luminance sig- 
nal Y tra obtained after gradation correction. For this rea- 
son, faithful color reproduction which keeps a hue can 
be performed without making an image monotonous as 
in a case using the same conversion coefficient as that 
used in calculation for a luminance. 
[01 04] When the conversion coefficient gain c is calcu- 
lated, color difference signals Cb org and Cr org obtained 
before correction are sequentially received from the Y/ 
C separation unit 1 2 (step S29) , and the color difference 
signals Cb^ and Cr^ obtained after correction are cal- 
culated as described in the following Equation 8 (step 
S33). 
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[Equation 8] 
Cb tra = gain c *Cb org 
Cr tra = gain c »Cr org 

[0105] The color difference signals Cb^ and Cr^ 
converted in this manner are output to the Y/C synthesis 
unit 19 (step S34). 

[01 06] In the Y/C synthesis unit 1 9, the luminance sig- 
nal Y^ obtained after gradation conversion and the 
color difference signals Cb tra and Cr^ obtained after 
conversion are Y/C-synthesized to be converted into, e. 
g., an RGB signal. The RGB signal is output through the 
output unit 15 under the control of the control unit 16. 
[0107] In the second embodiment, gradation correc- 
tion made in consideration of a theoretical limit charac- 
teristic of color reproduction is performed by an image 
processing apparatus constituted as a circuit in an elec- 
tronic camera. However, such these processes can be 
performed by a process program of a computer. 
[0108] This image processing apparatus is not only 
applied to an electronic camera, but also popularly ap- 
plied to devices such as printer devices which handle 
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images. 

[0109] According to the second embodiment, grada- 
tion correction made in consideration of a theoretical 
limit characteristic of color reproduction is performed to 
a color difference signal. For this reason, even though 
gradation correction of an image is performed, a hue can 
be kept. 

[0110] FIGS. 13 and 14 show the third embodiment 
of the present invention, wherein FIG. 13 is a block di- 
agram showing a basic configuration of an electronic 
camera, and FIG. 14 is a flow chart showing an image 
conversion process. 

[0111] The same reference numerals as in the first 
and second embodiments denote the same parts in the 
third embodiment, and a description thereof will be omit- 
ted. Only different points will be mainly described below. 
[01 12] The third embodiment is constituted such that 
a function obtained by combining the functions of the 
first embodiment and the second embodiment appears. 
More specifically, weighting depending on a photo- 
graphing condition is performed when a gradation con- 
version characteristic of a luminance signal is calculat- 
ed, a theoretical limit characteristic of color reproduction 
is considered when a conversion characteristic of a 
color difference signal is calculated, and a hue and a 
saturation are corrected depending on the photograph- 
ing condition. 

[0113] More specifically, an image signal output from 
the CCD 4 is converted into a digital signal by the A/D 
converter 5, and then the digital signal is temporarily 
stored in the image buffer 6. 

[0114] As described above, the photometric evalua- 
tion unit 7 and the focal point detection unit 8 outputs 
AE information and AF information to the control unit 16 
on the basis of image data accumulated in the image 
buffer 6 (step S41). 

[0115] On the other hand, the image data stored in 
the image buffer 6 are sequentially transmitted to the 
interpolation unit 10 and interpolated with respect to R 
image data, G image data, and B image data to be con- 
verted into three-CCD image data (step S42). The three- 
CCD image data are stored in the operation buffer 11 . 
[0116] In the Y/C separation unit 12, RGB image data 
is read from the operation buffer 11 , and the luminance 
signal Y and the color difference signals Cb and Cr are 
calculated as described in Equation 4 described above 
(step S43). 

[0117] The luminance correction unit 17, in the edge 
extraction unit 30, extracts an edge component from the 
luminance signal Y of these signals (step S44) s and a 
binarizing process is performed to the extracted edge 
component (step S45). 

[0118] On the other hand, in the photographing con- 
dition estimation unit 13, a photographing condition is 
estimated on the basis of the AF information and the AE 
information as described above (step S46), and one of 
Type 1 to Type 6 corresponding to a weight pattern is 
selected (step S47). A weight coefficient, as shown in 



FIGS. 6A to 6F, corresponding to the selected weight 
pattern is read from the weight pattern ROM 25 (step 
S48). 

[0119] In this manner, an edge histogram weighted by 
5 the histogram generation unit 31 is generated on the ba- 
sis of the edge component binarized in step S45 and a 
weight pattern read in step S8 (step S49), and an accu- 
mulated edge histogram is further generated from the 
edge histogram (step S50). 
10 [0120] The conversion curve calculation unit 32 cal- 
culates a gradation conversion curve on the basis of the 
accumulated edge histogram obtained in this manner 
(stepS51). 

[0121] In the subsequent luminance conversion unit 
is 33, the luminance signal Y output from the Y/C separa- 
tion unit 12 is subjected to a conversion process by the 
gradation conversion curve obtained from the conver- 
sion curve calculation unit 32 (step S52). The converted 
luminance signal Y is output to the color difference cor- 
20 rection unit 1 8 and output to the Y/C synthesis unit 1 9. 
[01 22] In the color difference correction unit 1 8, a first 
correction coefficient and a second correction coeffi- 
cient are calculated as described above on the basis of 
a luminance signal Y l obtained after the gradation cor- 
25 rection, the luminance signal Y obtained before the gra- 
dation correction and output from the Y/C separation 
unit 12, and a theoretical limit characteristic of color re- 
production read from the color reproduction limit char- 
acteristic ROM 36. A conversion coefficient to a color 
30 difference signal is calculated on the basis of the first 
correction coefficient and the second correction coeffi- 
cient. The color difference signals Cb and Cr received 
from the Y/C separation unit 12 are converted (step 
S53). 

35 [0123] Color difference signals Cb' and Cr* corrected 
by the color difference correction unit 1 8 are input to a 
skin color correction unit 41 serving as a second color 
difference correction means to perform correction orthe 
like of, e.g., a skin color with reference to a photograph- 
40 ing condition estimated by the photographing condition 
estimation unit 13 (step S54). 

[0124] In this case, when a portrait of a single person 
or plural persons as in Type 3 or Type 4 is used, a proc- 
ess of correcting a hue and a saturation is performed 
45 such that a skin color looks more preferable. For exam- 
ple, the skin color of Japanese is exemplified. That is, 
since Japanese tends to have a yellowish hue, a proc- 
ess of slightly shifting the yellowish hue to a reddish hue 
is performed. 

so [01 25] When a landscape containing sky on the upper 
side as in Type 1 , a process of correcting a hue and a 
saturation is performed such that the sky looks more 
blue. 

[0126] In addition, when a landscape is estimated as 
55 in Type 1 or Type 2, and when it is estimated on the 
basis of other information that the landscape contains a 
large number of plants, a process of correcting a hue 
and a saturation may be performed to make the green 
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of the plants preferable. The embodiment can be ap- 
plied to not only the above objects, but also various ob- 
jects. 

[0127] The color difference signals Cb' and Cr* cor- 
rected by the skin color correction unit 41 and the lumi- 
nance signal Y subjected to gradation correction by the 
luminance correction unit 17 are synthesized by the Y/ 
C synthesis unit 1-9 to be converted into an original RGB 
signal (step S55). Thereafter, the RGB signal is output 
through the output unit 15 (step S56). 
[0128] Also, in the third embodiment, the device op- 
erates a circuit in an electronic camera to perform gra- 
dation correction. However, these processes can also 
be performed by a process program of a computer. 
[0129] In this case, photometric information from the 
photometric evaluation unit 7 and focal information from 
the focal point detection unit 8 are added to the header 
portion of a file in which image data in the image buffer 
6 is stored, and the file is recorded on a recording me- 
dium such as a memory card. The file may be read by 
the computer to cause the computer to sequentially per- 
form necessary parts of the processes shown in FIG. 14. 
[0130] The process program is recorded on various 
recording media such as a hard disk in the computer, a 
portable floppy disk, and a photomagnetic disk. 
[0131] The image processing apparatus is not only 
applied to an electronic camera, but also popularly ap- 
plied to devices such as printer devices which handle 
images. 

[0132] According to the third embodiment described 
above, both of the effect of the first embodiment de- 
scribed above and the effect of the second embodiment 
described above can be achieved. At the same time, 
correction of a hue and a saturation can be performed 
depending on a photographing condition. For this rea- 
son, the skin color of a person which is, e.g., a main 
object can also be corrected into a preferable skin color. 
[01 33] FIGS. 1 5 to 1 8 show the fourth embodiment of 
the present invention, wherein FIG. 15 is a block dia- 
gram showing a basic configuration of an electronic 
camera. 

[01 34] The same reference numerals as in the first to 
third embodiments described above denote the same 
parts in the fourth embodiment, and a description there- 
of will be omitted. Only different points will be mainly de- 
scribed below. 

[0135] This embodiment is obtained by applying the 
image processing apparatus according to the present 
invention to an electronic camera. For the sake of de- 
scriptive convenience, a case in which two images, i.e., 
an image obtained by short-period exposure and an im- 
age obtained by iong-period exposure are synthesized 
with each otherto obtain one wide dynamic range image 
will be described below. As a matter of course, the em- 
bodiment can also be applied to a case in which a large 
number of images are synthesized with each other. 
[0136] This electronic camera, as shown in FIG. 15, 
has a configuration which is almost the same as that 



shown in FIG. 1 . However, since the electronic camera 
synthesizes a plurality of images, the electronic camera 
is different from the electronic camera shown in FIG. 1 
in the following points. 
5 [0137] More specifically, as image buffers for accu- 
mulating image data of one screen digitized by the A/D 
converter 5, a first image buffer 6a for storing the image 
obtained by long-period exposure and a second image 
buffer 6b for storing the image obtained by short-period 
10 exposure are arranged. 

[01 38] The photometric evaluation unit 7 reads image 
data from the first image buffer 6a also used to accumu- 
late data for a photometric operation and focal point de- 
tection to calculate a luminance distribution. The aper- 
15 ture diameter of the diaphragm 2 and the electronic 
shutter of the CCD 4 are controlled such that an appro- 
priate exposure is obtained in a photographing state. 
[0139] In addition, the focal point detection unit 8 
reads image data from the first image buffer 6a to detect 
20 a focal position. On the basis of the detection result, the 
AF motor 9 is controlled. 

[0140] The interpolation unit 10 interpolates single- 
CCD image data read from the first and second image 
buffers 6a and 6b to convert the image data into three- 
25 CCD image data. 

[0141] The electronic camera further comprises: an 
appropriate exposure extraction unit 51 serving as an 
extraction means for reading the luminance signal Y 
from the Y/C separation unit 12 to decide, by a signal 
30 level, whether each of the pixels constituting the entire 
screen has an appropriate exposure or not, and for, on 
the basis of the result, extracting and outputting division- 
al image information; a conversion characteristic calcu- 
lation unit 52 serving as a gradation correction means 
35 for performing weighting when a histogram of an edge 
which is a characteristic amount with reference the es- 
timation result obtained by the photographing condition 
estimation unit 13 to calculate a conversion character- 
istic and for performing gradation conversion on the ap- 
40 propriate exposure region output from the appropriate 
exposure extraction unit 51 ; and an image synthesis unit 
53 serving as a synthesis means for synthesizing an im- 
age related to long-period exposure and an image re- 
lated to short-period exposure which are obtained after 
45 gradation conversion and output from the conversion 
characteristic calculation unit 52 with reference to region 
information output from the appropriate exposure ex- 
traction unit 51 to generate one wide dynamic range im- 
age. A wide dynamic range image synthesized by the 
so image synthesis unit 53 is output by the output unit 15 
to, e.g., a recording medium or a display device. 
[01 42] In addition, the control unit 1 6 performs control 
of the entire electronic camera, and also controls the ap- 
propriate exposure extraction unit 51, the conversion 
55 characteristic calculation unit 52, and the image synthe- 
sis unit 53, as a matter of course. 
[0143] FIG. 1 6 is a block diagram showing a detailed 
configuration of the photographing condition estimation 
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unit 13. The internal configuration in FIG. 16 is the same 
as that shown in FIG. 2. The internal configuration in 
FIG. 16 is different from the internal configuration in FIG. 
2 only that results classified into the types described 
above are output from the integration, unit 22 to the con- 
version characteristic calculation unit 52. 
[0144] Subsequently, FIG. 17 is a block diagram 
showing a detailed conf iguration of the conversion char- 
acteristic calculation unit. 

[0145] The internal configuration of the conversion 
characteristic calculation unit 52 is almost the same as 
that of the gradation correction unit 14 (see FIG. 3) in 
the fist embodiment described above. 
[0146] More specifically, as described above, the ap- 
propriate exposure extraction unit 51 reads the lumi- 
nance signal Y of a long-period exposure image and 
compares the signal level of each of pixels constituting 
the entire screen with a predetermined value to decide 
whether the corresponding pixel has an appropriate ex- 
posure or not. A set of pixels which are determined as 
pixels having appropriate exposures corresponds to an 
appropriate exposure region related to long-period ex- 
posure, and the other portion except for the appropriate 
exposure region related to the long-period exposure 
corresponds to an appropriate exposure region related 
to short-period exposure. 

[0147] When the luminance signal Y in the appropri- 
ate exposure region of the long-period exposure output 
from the appropriate exposure extraction unit 51 is input 
to the edge detection unit 26 serving as a characteristic 
amount calculation means, the edge detection unit 26 
performs edge detection under the control of the control 
unit 16. 

[0148] The other components, i.e., the pattern selec- 
tion unit 24, the weight pattern ROM 25, the histogram 
generation unit 27, and the conversion curve calculation 
unit 28 have a configuration which is almost the same 
as that of the first embodiment described above to be 
operated. 

[0149] In addition, the conversion unit 29 serving as 
a conversion means is the same as that in the first em- 
bodiment except that image data is input from the ap- 
propriate exposure extraction unit 51 , and the image da- 
ta obtained after gradation correction is output to the im- 
age synthesis unit 53. In the conversion unit 29, grada- 
tion correction of the luminance signal Y related to long- 
period exposure is performed, and gradation correc- 
tions of the color difference signals Cb and Cr related to 
the long-period exposure are sequentially performed to 
output the corrected signals to the image synthesis unit 
53. Thereafter, gradation corrections of a luminance sig- 
nal Y and color difference signals Cb and Cr related to 
short-period exposure are performed in the same man- 
ner as described above to output the corrected signals 
to the image synthesis unit 53. 

[0150] The subsequent image synthesis unit 53 re- 
ceives the luminance signal Y and the color difference 
signals Cb and Cr obtained after the gradation correc- 



tion related to the long-period exposure to generate, e. 
g., an RGB signal related to the long-period exposure. 
The image synthesis unit 53 receives the luminance sig- 
nal Y and the color difference signals Cb and Cr ob- 
5 tained after the gradation correction related to the short- 
period exposure to generate the RGB signal related to 
the short-period exposure. Thereafter, these signals are 
synthesized with each other to generate a wide dynamic 
range image, and the image is output. 
10 [0151] FIG. 18 is a flowchart showing an image con- 
version process . 

[01 52] An object image focused on the CCD 4 consti- 
tuted by a single CCD is designed to perform image 
pickup operations plural times under different exposure 
15 conditions. As described above, an image pickup oper- 
ation by long-period exposure and an image pickup op- 
eration by short-period exposure are performed in this 
order to sequentially output signals as image signals. 
[01 53] These image signals are converted into digital 
20 signals by the A/D converter 5. Thereafter, the digital 
signals are stored in the first image buffer 6a and the 
second image buffer 6b, respectively. 
[01 54] The photometric evaluation unit 7 and the focal 
point detection unit 8 outputs AE information and AF in- 
25 formation to the control unit 1 6 as described above on 
the basis of image data of long-period exposure accu- 
mulated in the first image buffer 6a which is one of these 
buffers (step S61). 

[0155] On the other hand, the image data stored in 
30 the first image buffer 6a and the second image buffer 6b 
are sequentially transmitted to the interpolation unit 10 
and interpolated with respect to R image data, G image 
data, and B image data to be converted into three-CCD 
image data (step S62). The three-CCD image data are 
35 stored in the operation buffer 11 . 

[01 56] In the Y/C separation unit 1 2, RGB image data 
is read from the operation buffer 1 1 , and the luminance 
signal Y and the color difference signals Cb and Cr are 
calculated as described in Equation 4 described above 
40 (step S63). 

[0157] The appropriate exposure extraction unit 51 
compares the signal level of the luminance signal Y of 
these signals with a predetermined threshold value with 
respect to pixels to decide whether the corresponding 
45 pixel belongs an appropriate exposure region or not, so 
that divisional image information Is extracted and output 
(step S64). 

[0158] Thereafter, the edge detection unit 26 in the 
conversion characteristic calculation unit 52 operates a 
so known quadratic differential filter such as a Laplacian to 
the luminance signal Y to extract an edge component 
/step S65). A threshold value which is about twice a 
standard deviation is set for the extracted edge compo- 
nent to perform a binarizing process (step S66). 
55 [0159] On the other hand, in the photographing con- 
dition estimation unit 13, the above photographing con- 
dition is estimated on the bas is of the AF information 
and the AE information (step S67), and one of Type 1 
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to Type 6 corresponding to a weight pattern is selected 
as shown in FIG. 5 (step S68). A weight coefficient cor- 
responding to the selected weight pattern as shown in 
FIGS. 6A to 6F is read from the weight pattern ROM 25 
(step S69). 

[0160] In this manner, the histogram generation unit 
27 generates an edge histogram weighted by the histo- 
gram generation unit 27 on the basis of the edge com- 
ponent binarized in stepS66 and the weight pattern read 
in step S69"(step S70), and an accumulated edge his- 
togram is generated from the edge histogram (step 
S71). 

[0161] On the basis of the accumulated edge histo- 
gram obtained as described above, the conversion 
curve calculation unit 28 calculates a gradation conver- 
sion curve (step S72). 

[0162] In the subsequent conversion unit 29, the lu- 
minance signal Y and the color difference signals Cb 
and Cr output from the appropriate exposure extraction 
unit 51 are subjected to a conversion process by the gra- 
dation conversion curve obtained from the conversion 
curve calculation unit 28 (step S73), and the converted 
image data is output (step S74). 

[0163] In the fourth embodiment, gradation correction 
corresponding to a photographing condition is per- 
formed by an image processing apparatus constituted 
as a circuit in an electronic camera. However, these 
processes can also be performed by a process program 
of a computer. In this case, photographing information 
such as photometric information and focal information 
is recorded on, e.g., the header portion of an image file. 
In the computer, a photographing condition may be es- 
timated on the basis of these pieces of photographing 
information to perform gradation correction appropriate 
to the photographing condition. 

[0164] The image processing apparatus is not only 
applied to an electronic camera, but also popularly ap- 
plied to devices such as printer devices which handle 
images. 

[0165] According to the fourth embodiment described 
above, as in the embodiments described above, grada- 
tion correction which is most appropriate to a photo- 
graphing scene can be performed in consideration of a 
main object. 

[0166] FIGS. 19 to 21 show the fifth embodiment of 
the present invention, wherein FIG 19 is a block dia- 
gram showing a basic configuration of an electronic 
camera, FIG. 20 is a block diagram showing a detailed 
configuration of a luminance correction unit, and FIG. 
21 is a block diagram showing a detailed configuration 
of a color difference correction unit. 
[0167] The same reference numerals as in the first to 
fourth. embodiments described above denote the same 
parts in the fifth embodiment, and a description thereof 
will be omitted. Only different points will be mainly de- 
scribed below. 

[0168] In the fifth embodiment, a luminance signal Y 
and color difference signals Cb and Cr separated by the 



Y/C separation unit 12, as shown in FIG. 19, are input 
to the luminance correction unit 17 serving as a grada- 
tion correction means and the color difference correc- 
tion unit 18 serving as a color difference correction 
5 means through the appropriate exposure extraction unit 
51 , respectively. 

[01 69] The luminance correction unit 1 7, as shown in 
FIG. 20, has the same configuration as that shown in 
FIG. 9 except that the luminance signal Y of an appro- 
10 priate exposure region is received from the appropriate 
exposure extraction unit 51 . The processes of the lumi- 
nance correction unit 17 are the same as those shown 
in FIG. 12. 

[0170] In addition, the color difference correction unit 
15 1 8, as shown in FIG. 21 , has the same configuration as 
thait shown in FIG. 1 0 except that the first correction co- 
efficient calculation unit 37 and the color difference con- 
version unit 38 receive outputs from the appropriate ex- 
posure extraction unit 51 . 
20 [01 71 ] In the Y/C synthesis unit 1 9, a luminance signal 
Y tra obtained after gradation conversion and color dif- 
ference signals Cb tra , Cr^ obtained after conversion 
are Y/C-synthesized with each other to be converted in- 
to, e.g., an RGB signal, and the RGB signal is output to 
25 the image synthesis unit 53, 

[0172] in the image synthesis unit 53, an appropriate 
exposure image portion obtained after gradation con- 
version related to long-period exposure and an appro- 
priate exposure image portion obtained after gradation 
30 conversion related to short-period exposure are synthe- 
sized with each other to generate a wide dynamic range 
image. Thereafter, the wide dynamic range image is out- 
put from the output unit 15. 

[0173] In this fifth embodiment, gradation correction 
35 made in consideration of a theoretical limit characteristic 
of color reproduction is performed by an image process- 
ing apparatus constituted as a circuit in an electronic 
camera. However, these processes can also be per- 
formed by a process program of a computer. 
40 [0174] The image processing apparatus is not only 
applied to an electronic camera, but also popularly ap- 
plied to devices such as printer devices which handle 
images. 

[0175] According to the fifth embodiment, gradation 
45 correction made in consideration of a theoretical limit 
characteristic of color reproduction is performed to a 
color difference signal. For this reason, even though 
gradation correction of an image is performed, an ap- 
propriate hue can be kept. 
so [0176] FIG. 22 shows the sixth embodiment of the 
present invention, and is a block diagram showing a ba- 
sic configuration of an electronic camera. 
[0177] The same reference numerals as in the first to 
fifth embodiments denote the same parts in the sixth 
55 embodiment, and a description thereof will be omitted. 
Only different points will be mainly described below. 
[01 78] The sixth embodiment is constituted such that 
a function obtained by combining the functions of the 
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fourth embodiment and the fifth embodiment appears. 
More specifically, weighting depending on a photo- 
graphing condition is performed when a gradation con- 
version characteristic of a luminance signal is calculat- 
ed, a theoretical limit characteristic of color reproduction 
is considered when a conversion characteristic of a 
color difference signal is calculated. 
[0179] More specifically, as shown in FIG. 22, the 
classification results of photographing conditions as 
shown in FIG. 5 estimated by the photographing condi- 
tion estimation unit 13 are input to the luminance cor- 
rection unit 1 7. When an edge histogram of a luminance 
signal is generated in the luminance correction unit 17, 
weighting as shown in FIGS. 6A to 6F is performed. 
[0180] When a luminance signal subjected to grada- 
tion conversion on the basis of the characteristic curve 
obtained as described above is input to the color differ- 
ence correction unit 1 8, as in the fifth embodiment de- 
scribed above, a second correction coefficient is calcu- 
lated with reference to a table showing the theoretical 
limit characteristic of color reproduction. Similarly, a first 
correction coefficient is calculated on the basis of a lu- 
minance signal obtained before gradation conversion. 
On the basis of the first correction coefficient and the 
second correction coefficient, a conversion coefficient 
related to a color difference signal is calculated. Con- 
version appropriate to a color difference is performed to 
output the converted coefficient to the Y/C synthesis unit 

19 - 

[0181] In the Y/C synthesis unit 19. a luminance signal 
obtained after gradation conversion and output from the 
luminance correction unit 17 and a color difference sig- 
nal obtained after conversion and output from the color 
difference correction unit 18 are Y/C-synthesized with 
each other to output the synthesized signal to the image 
synthesis unit 53. In the image synthesis unit 53, an ap- 
propriate exposure image portion obtained after grada- 
tion conversion related to long-period exposure and an 
appropriate exposure image portion obtained after gra- 
dation conversion related to short-period exposure are 
synthesized with each other to generate a wide dynamic 
range image. The wide dynamic range image is output 
from the output unit 15. 

[0182] Also in the sixth embodiment, gradation cor- 
rection is performed by an image processing apparatus 
constituted as a circuit in an electronic camera. Howev- 
er, these processes can also be performed by a process 
program of a computer. 

[0183] The image processing apparatus is not only 
applied to an electronic camera, but also popularly ap- 
plied to devices such as printer devices which handle 

images. 

[0184] According to the sixth embodiment described 
above, both of the effect of the fourth embodiment de- 
scribed above and the effect of the fifth embodiment de- 
scribed above can be achieved. 

[0185] In this invention, it is apparent that working 
modes different in a wide range can be formed on this 



basis of this invention without departing from the spirit 
and scope of the invention. This invention is not restrict- 
ed by any specific embodiment except being limited by 
the appended claims. 



Claims 

1. An image processing apparatus characterized by 
adjusting a gradation range of an input image, com- 
prising: 

photographing condition estimation means for 
estimating a photographing condition of the in- 
put image; and 

gradation correction means for performing gra- 
dation correction on the input image on the ba- 
sis of the photographing condition to adjust the 
gradation range to a predetermined gradation 
range. 

2. An image processing apparatus characterized by 
adjusting a gradation range of an input image, com- 
prising: 

luminance-color difference separation means 
for separating the input image into a luminance 
signal and a color difference signal; 
gradation correction means for performing gra- 
30 dation correction on the luminance signal to ad- 

just the gradation range to a predetermined 
gradation range; 

color difference correction means for correcting 
the color difference signal on the basis of a lu- 
35 minance signal obtained before the gradation 

correction and output from the luminance-color 
difference separation means, a luminance sig- 
nal obtained after the gradation correction and 
output from the gradation correction means, 
40 and a theoretical limit characteristic of color re- 

production; and 

luminance-color difference synthesis means 
for synthesizing the luminance signal obtained 
after the gradation correction and a color differ- 
45 ence signal obtained after the correction into an 

image signal. 



25 



3. An image processing apparatus characterized by 
adjusting a gradation range of an input image, corn- 
so prising: 

photographing condition estimation means for 
estimating a photographing condition of the in- 
put image; 

55 luminance-color difference separation means 

for separating the input image into a luminance 
signal and a color difference signal; 
gradation correction means for performing gra- 
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dation correction of the luminance signal on the 
basis of the photographing condition to adjust 
the gradation range to a predetermined grada- 
tion range; 

color difference correction means for perform- 5 
ing correction of the color difference signal on 
the basis of a luminance signal obtained before 
the gradation correction and output from the lu- 
mihance-color difference separation means, a 
luminance signal obtained after gradation cor- 10 
rection and output from the gradation correction 
means, and a theoretical limit characteristic of 
color reproduction; and 
luminance-color difference synthesis means 
for synthesizing an image signal on the basis is 
of the luminance signal obtained after the gra- 
dation correction and a color difference signal 
obtained after the correction. 6. 

An image processing apparatus characterized by 20 
processing an image group constituted by a plural- 
ity of images obtained by picking up the same object 
under different exposure conditions to generate one 
wide dynamic range image, comprising: 

25 

photographing condition estimation means for 
estimating a photographing condition; 
extraction means for extracting an appropriate 
exposure region on the basis of an image signal 
level of each image in the image group; 30 
gradation correction means for performing gra- 
dation correction on the appropriate exposure 
region on the basis of the photographing con- 
dition; and 

synthesis means for synthesizing appropriate 35 
exposure regions subjected to gradation cor- 
rection by the gradation correction means to 
generate one wide dynamic range image. 

An image processing apparatus characterized by *o 
processing an image group constituted by a plural- 
ity of images obtained by picking up the same object 
under different exposure conditions to generate one 
wide dynamic range image, comprising: 

45 

luminance-color difference separation means 
for separating an image signal of each image 
in the image group into a luminance signal and 
a color difference signal; 

extraction means for extracting an appropriate s° 
exposure region on the basis of a signal level 
of the luminance signal; 

gradation correction means for performing gra- 
dation correction on a luminance signal of the 
appropriate exposure region; 55 
color difference correction means for perform- 
ing correction on a color difference signal of the 
appropriate exposure region on the basis of a 



luminance signal obtained before the gradation 
correction and output from the luminance-color 
difference separation means, a luminance sig- 
nal obtained after gradation correction and out- 
put from the gradation correction means, and a 
theoretical limit characteristic of color repro- 
duction; 

luminance-color difference synthesis means 
for synthesizing an image signal on the basis 
of the luminance signal obtained after the gra- 
dation correction and a color difference signal 
obtained after the correction; and 
synthesis means for synthesizing the synthe- 
sized image signal of the appropriate exposure 
regions to generate one wide dynamic range 
image. 

An image processing apparatus characterized by 
processing an image group constituted by a plural- 
ity of images obtained by picking up the same object 
under different exposure conditions to generate one 
wide dynamic range image, comprising: 

photographing condition estimation means for 
estimating a photographing condition; 
luminance-color difference separation means 
for separating an image signal of each image 
in the image group into a luminance signal and 
a color difference signal; 
extraction means for extracting an appropriate 
exposure region on the basis of a signal level 
of the luminance signal; 
gradation correction means for performing gra- 
dation correction on the luminance signal of the 
appropriate exposure region on the basis of the 
photographing condition; 
color difference correction means for perform- 
ing correction on a color difference signal of the 
appropriate exposure region on the basis of a 
luminance signal obtained before the gradation 
correction and output from the luminance-color 
difference separation means, a luminance sig- 
nal obtained after gradation correction and out- 
put from the gradation correction means, and a 
theoretical limit characteristic of color repro- 
duction; 

luminance-color difference synthesis means 
for synthesizing an image signal on the basis 
of the luminance signal obtained after the gra- 
dation correction and a color difference signal 
obtained after the correction; and 
synthesis means for synthesizing the synthe- 
sized image signal of the appropriate exposure 
regions to generate one wide dynamic range 
image. 

7. An image processing apparatus according to claim 
1 , 3, 4, or 6, characterized in 
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the photographing condition estimation 
means estimates a photographing condition on the 
basis of at least one piece of information of focal 
information, photometric information, zoom position 
information, multi-spot photometric information, 
line-of-sight input information, and strobe flash in- 
formation. 

8. An image processing apparatus according to claim 
1,3,4, or 6, characterized in 

the photographing condition estimation 
means comprises: 

focal position estimation means for estimating 
at least three types of focal positions including 
a scenic photographing operation, a portraiture 
photographing operation, and a close-up pho- 
tographing operation from the focal informa- 
tion; 

object distribution estimation means for esti- 
mating at least three types of object distribu- 
tions of an entire screen, a center focus, and a 
central portion from the photometric informa- 
tion; and 

integration means for integrally estimating a 
photographing condition by combining the focal 
position estimated by the focal position estima- 
tion means and the object distribution estimat- 
ed by the object distribution estimation means. 

9. An image processing apparatus according to claim 
1,3,4, or 6, characterized in 

the gradation correction means comprises: 

selection means for selecting an arrangement 
of a weight coefficient on the basis of the pho- 
tographing condition; 

characteristic amount calculation means for 
calculating a characteristic amount with respect 
to the input image; 

histogram generation means for generating a 
weighting histogram related to the characteris- 
tic amount on the basis of the arrangement of 
the weight coefficient; 

gradation conversion curve calculation means 

for calculating a gradation conversion curve on 

the basis of the histogram; and 

conversion means for performing gradation 

conversion by using the gradation conversion 

curve. 
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histogram generation means for generating a 
histogram related to the characteristic amount; 
gradation conversion curve calculation means 
for calculating a gradation conversion curve on 
the basis of the histogram; and 
luminance conversion means for performing 
gradation conversion of a luminance signal by 
using the gradation conversion curve. 

11. An image processing apparatus according to claim 
2, 3, 5, or 6, characterized in 

the color difference correction means com- 
prises: 

first calculationmeans for calculating a first cor- 
rection coefficient on the basis of a luminance 
signal obtained before the gradation correction 
and the theoretical limit characteristic of color 
reproduction; 

second calculation means for calculating a sec- 
ond correction coefficient on the basis of the lu- 
minance signal obtained after the gradation 
correction and the theoretical limit characteris- 
tic of color reproduction; and 
color difference conversion means for convert- 
ing a color difference signal by using the first 
correction coefficient and the second correction 
coefficient. 
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An image processing apparatus according to claim 
3 or 6, further comprising: 

second color difference correction means for 
correcting at least one of a hue and a saturation of 
a specific color in the color difference signal ob- 
tained after the correction on the basis of the pho- 
tographing condition. 



10. An image processing apparatus according to claim 
2 or 5, characterized in 

the gradation correction means comprises: 

characteristic amount calculation means for 
calculating a characteristic amount related to 
the luminance signal; 



13. An image processing apparatus according to claim 
1 , 2, or 3, characterized in 

40 the gradation correction means adjusts a gra- 

dation range by reducing the gradation range. 

14. A recording medium characterized by recording a 
process program, executed by a computer, for ad- 

45 justing a gradation range of an input image, the 
process program comprising: 

the step of separating the input image into a lu- 
minance signal and a color difference signal; 
so the step of performing gradation correction on 

the luminance signal to adjust the gradation 
range to a predetermined gradation range; 
the step of performing correction on the color 
difference signal on the basis of a luminance 
55 signal obtained before the gradation correction, 

a luminance signal obtained after the gradation 
correction, and a theoretical luminance charac- 
teristic of color reproduction; and 
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the step of synthesizing an image signal on the 
basis of the luminance signal obtained after 
gradation correction and a color difference sig- 
nal obtained after the correction. 

15. A recording medium characterized by recording a 
process program, executed by a computer, for 
processing an image group constituted by a plural- 
ity of images obtained by image pickup performed 
to the same object under different exposure condi- 
tions to generate one wide dynamic range image, 
the process program comprising: 

the step of separating an image signal of each 
image in the image group into a luminance sig- 
nal and a color difference signal; 
the step of extracting an appropriate exposure 
region on the basis of a signal level of the lumi- 
nance signal; 

the step of performing gradation correction on 
a luminance signal of the appropriate exposure 
region; 

the step of performing correction on a color dif- 
ference signal of the appropriate exposure re- 
gion on the basis of a luminance signal ob- 25 
tained before the gradation correction, a lumi- 
nance signal obtained after gradation correc- 
tion, and a theoretical limit characteristic of 
color reproduction; 

the step of synthesizing an image signal on the 
basis of the luminance signal obtained after the 
gradation correction and a color difference sig- 
nal obtained after the correction; and 
the step of synthesizing the synthesized image 
signal of the appropriate exposure regions to 
generate one wide dynamic range image. 
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